Size Change: +40.4 kB (+3%)

Total Size: 1.44 MB

_{compressed-size-action}

Flaky tests detected in e77d0b7.
Some tests passed with failed attempts. The failures may not be related to this commit but are still reported for visibility. See the documentation for more information.

🔍 Workflow run URL: https://github.com/WordPress/gutenberg/actions/runs/5345949860
📝 Reported issues:

• [Flaky Test] shouldn't conflict with server-side autosave #35176 in specs/editor/various/autosave.test.js

I've prepared a Playwright vs. Puppeteer metrics comparison by running all the specs against both frameworks in a single CI job. Here are the results:

post-editor

site-editor

front-end-classic-theme

front-end-block-theme

The biggest difference seems to be around the initial painting area (FP, FCP, LCP), which might be caused by the initialization of the custom e2e utils for each page, as mentioned by @kevin940726 in #51084 (comment). I'd like to try going vanilla Playwright to see if this is true, but I'd leave it for a follow-up PR. For this PR, I'd leave Puppeteer running perf specs as they are and only merge the migrated ones. Once we agree that the numbers are 👌, we do the switch. How does that sound, @youknowriad @oandregal @dmsnell?

@WunderBart thanks again for all the work you've put into this. I'm afraid I don't have much of value to offer, other than I don't see any reason this shouldn't be done. it's obvious the metrics will change; which I figure is the tradeoff you see as valuable enough to accept in order to eliminate Puppeteer from the repo. That's something you have more context about than I do, so it sounds fine enough.

I'll add my approval which is just that: me voicing my thoughts and nothing more. Please lean on whatever other review you see fitting, as well as your own vetting and testing.

There is some confusion at the moment with existing discrepancies in testing metrics. After speaking with @youknowriad it seems like this could be a poor time to change them even more dramatically until we all know what happened and how to fix consistency. For background, it appears like changing the base version against which performance numbers are compared has skewed the measurements, or led to reporting slower performance when there's suspicion that it isn't slower.

All this is to say I pulled my approval back because we need to figure these things out first.

@WunderBart have you considered the impact on release performance reporting? Have you thought about running the tests against old versions to establish new baselines for comparing performance?

There is some confusion at the moment with existing discrepancies in testing metrics. After speaking with @youknowriad it seems like this could be a poor time to change them even more dramatically until we all know what happened and how to fix consistency.

Possibly this was the reason for the skewed metrics? #51407 (comment)

For background, it appears like changing the base version against which performance numbers are compared has skewed the measurements, or led to reporting slower performance when there's suspicion that it isn't slower.

All this is to say I pulled my approval back because we need to figure these things out first.

@WunderBart have you considered the impact on release performance reporting? Have you thought about running the tests against old versions to establish new baselines for comparing performance?

Yes, although for this PR, I only wanted to do the 1:1 migration (no improvements, do with Playwright the same thing we do with Puppeteer) and merge without switching to Playwright. In the follow-up PRs, I wanted to address the aforementioned inconsistencies and improvements and collect all the measurements necessary for the switch.

Right now, the 1:1 migration is done, and the code is working. Merging this PR would not affect the current performance job at all.

Thanks for all the work on this PR. Just want to make sure that we're still producing the same files as output so we continue to log things properly. Other than that I'm fine merging this.

I think after a dozen commits, we'll get a clear idea about whether this is more stable or less stable than before.

edit: I accidentally deleted the comment instead of just editing it, so reposting now. Sorry for the noise!

Below are the results of 100 rounds of the Puppeteer vs. Playwright perf comparison job for the post editor suite. A hundred rounds were way too much for CI (which is how I learned about the 6h job timeout), so I decided to run it locally overnight. Click here for the source sheet.

Regarding the results, one thing that can be noticed immediately is that Puppeteer shows more stability in the load aspect, while Playwright is definitely better with the UI. Some metrics also show significant yet regular bumps (inserterOpen, listViewOpen, focus) for both frameworks which we might want to even out (e.g., by dropping the warmup sample).

I'd like to move forward with this PR (uninvasively), so please let me know what you think, folks! 🙌

/cc @dmsnell @youknowriad @oandregal @kevin940726

Loading performance

Metric	Puppeteer	Playwright
serverResponse
domContentLoaded
loaded
firstPaint
firstContentfulPaint
firstBlock

UI performance

Metric	Puppeteer	Playwright
type
typeContainer
inserterOpen
inserterHover
inserterSearch
listViewOpen
focus

I defer to @youknowriad and the aforementioned issue with the base version (which I think might already be resolved) - as far as my review is concerned this seems fine to merge.

Thanks for all the work on this PR. Just want to make sure that we're still producing the same files as output so we continue to log things properly. Other than that I'm fine merging this.

Cool! This PR's code is totally passive - it doesn't change anything in the performance metrics area, just adds the migrated tests so that we have a starting point.

As per #51084 (comment) and #51084 (comment), this one's safe to merge as it doesn't yet switch to Playwright. The remaining suggestions will be addressed in a follow-up PR so there's a clear distinction between the migration and enhancements. /cc @kevin940726